The present invention relates generally to computing systems, and specifically to systems that use packet-switching fabrics to connect a computer host to peripheral devices.
In current-generation computers, the central processing unit (CPU) is connected to the system memory and to peripheral devices by a parallel bus, such as the ubiquitous Peripheral Component Interface (PCI) bus. PCI bus protocols and rules are described in detail in the PCI Local Bus Specification, Revision 2.2 (1998), published by the PCI Special Interest Group (Hillsboro, Oreg.), which is incorporated herein by reference. Briefly, the PCI specification defines a set of bus commands, or cycles, each identified by a 4-bit code on the C/BE (Bus Command and Byte Enable) lines of the bus. A bus master, such as a CPU, sends commands to a target device on the bus by specifying a 32- or 64-bit bus address of the target device. Table I below lists the PCI bus command codes for reference. The different types of read and write commands in the table are based on the division of the overall PCI address space into Memory, I/O and Configuration address spaces.
As data path-widths grow, and clock speeds become faster, the parallel bus is becoming too costly and complex to keep up with system demands. In response, the computer industry is moving toward fast, packetized, serial input/output (I/O) bus architectures, in which computing hosts and peripheral are linked by a switching network, commonly referred to as a switching fabric. A number of architectures of this type have been proposed, including xe2x80x9cNext Generation I/Oxe2x80x9d (NGIO) and xe2x80x9cFuture I/Oxe2x80x9d (FIO), culminating in the xe2x80x9cInfiniBandxe2x80x9d architecture, which has been advanced by a consortium led by a group of industry leaders (including Intel, Sun, Hewlett Packard, IBM, Compaq, Dell and Microsoft). Storage Area Networks (SAN) provide a similar, packetized, serial approach to high-speed storage access, which can also be implemented using an InfiniBand fabric.
Communications between a parallel bus and a packet network generally require a communications interface, to convert bus cycles into appropriate packets and vice versa. For example, a host channel adapter or target channel adapter can be used to link a parallel bus, such as the PCI bus, to the InfiniBand fabric. When the adapter receives data from a device on the PCI bus, it inserts the data in the payload of an InfiniBand packet, and then adds an appropriate header and error checking code, such as a cyclic redundancy check (CRC) code, as required for network transmission. The InfiniBand packet header includes a routing header and a transport header. The routing header contains information at the data link protocol level, including fields required for routing the packet within and between fabric subnets. The transport header contains higher-level, end-to-end transport protocol information. Similar headers are used in other types of packet networks known in the art, such as Internet Protocol (IP) networks.
Similarly, when the channel adapter receives a packet over the fabric for delivery to a device on the bus, it strips off the header and then generates an appropriate bus cycle, to convey the packet payload to the bus device. As a rule, the adapter is designed so that applications running on a CPU on the bus do not have to deal with network protocol considerations, such as generating data link or routing information and computing headers, and likewise so that applications running on a network host do not have to deal with bus cycle generation.
In contrast to interface device and methods known in the art, it is an object of some aspects of the present invention to provide a network interface that enables an application running on a bus device to access and program all parts of the header, as well as the payload, of a packet for transmission over a packet network.
It is a further object of some aspects of the present invention to provide a network interface that enables an application running on a network device to directly generate cycles on a parallel bus.
In some preferred embodiments of the present invention, a bridge device links a parallel bus, such as a PCI bus, to a packet network, such as an InfiniBand fabric. The bridge device comprises inbound and outbound packet registers, having addresses in an address space of the parallel bus, so that a master device on the bus, such as a CPU, can read and write to the registers. A software application running on the master device writes the entire substantive contents of a packet to the outbound packet register, including both the payload and the header, addressed to a target device on the network. The application is programmed so that the packet will conform with all of the relevant network protocols. The master device notifies the bridge device when it has finished writing the packet. The bridge device then transmits the packet over the network as is, preferably after having added a suitable error checking code. After the packet is sent, the outbound packet register is cleared.
The target device on the network can similarly send an inbound packet to the master device on the bus. The packet is preferably addressed to a management agent running on the bridge device, which places the packet data in the inbound packet register. The bridge device then generates an interrupt to the master device, notifying it that a packet has arrived. The application running on the master device reads the inbound packet register, which is then cleared and prepared to receive further packets.
In one of these preferred embodiments, the inbound and outbound packet registers are used to create a xe2x80x9cvirtual componentxe2x80x9d on the bus, whose effective bus address is the address of the registers. When the master device writes to this bus address, the bridge device hardware generates a packet to the specified target device. If this target device is an embedded processor, in another bridge device on the network, for example, the packet may elicit certain behaviors at the destination, such as assertion of interrupts or other external signals.
In further preferred embodiments of the present invention, the bridge device comprises a bus cycle register, additionally or alternatively to the inbound and outbound packet registers mentioned above. The bus cycle register is accessible for writing by devices on the network at a virtual network address that is associated with the bridge device. A software application running on a host device on the network sends a command packet to the bridge device at the network address, instructing the bridge device to write the bus address of a target device on the bus, as well as data to be delivered to the target device, into the bus cycle register. The application also writes an appropriate command code, such as a PCI C/BE code, to a designated field of the bus cycle register. The host device notifies the bridge device when it has finished writing to the bus cycle register, whereupon the bridge device generates a bus cycle with the command code, address and data specified by the host device.
Preferred embodiments of the present invention thus differ fundamentally from bridges known in the art, which attempt to make the bus/network interface transparent to devices on either side of the Interface insofar as possible. Transparent bridge implementations of this sort are described, for example, in two U.S. patent applications, by Kagan et al., entitled xe2x80x9cBridge between Parallel Buses over a Packet-Switched Networkxe2x80x9d and xe2x80x9cParallel Bus Communications over a Packet-Switching Fabric.xe2x80x9d Both of these applications are assigned to the assignee of the present patent application and are incorporated herein by reference. In contrast to the present invention, these applications describe bridge devices that translate bus cycles into complete network packets, and vice versa, so that devices on the bus are relieved entirely of the need to deal with network protocols, and network devices are relieved of having to deal with bus cycles. Although this transparent translation model is useful in most software applications that use the bridge, there are some applications, such as network management applications, in which it is desirable for a processor on the bus to be able to read and write packets to the network directly, without the channel adapter as intermediary. Bus and system management applications running on the network may similarly need to generate certain bus cycles. The preferred embodiments described herein provide a novel solution to these needs.
There is therefore provided, in accordance with a preferred embodiment of the present invention, a bridge device, for coupling a parallel bus to a packet network, the device including:
a bus interface adapter, coupled to the parallel bus so as to receive bus cycles from a master device on the bus;
an outbound packet register, having a bus address in an address space of the bus and adapted to store an outbound network address header and payload data written to the bus address of the register by the master device in one or more of the bus cycles received by the bus interface adapter; and
a network interface adapter, coupled to the network so as to transmit over the network an outbound packet containing the outbound network address header and payload data from the register, to a target device on the network specified by the network address header.
Preferably, the network address header includes a network routing header, wherein the network interface adapter is adapted to transmit the packet substantially without computation of network routing information for the packet beyond that contained in the network address header written by the master device.
Additionally or alternatively, the network interface adapter is adapted to compute and append an error checking code to the packet.
Preferably, the device includes a register status indicator, which is set to indicate that the entire network address header and payload data have been written to the outbound packet register, wherein the network interface adapter transmits the packet responsive to setting the status indicator.
Further preferably, the bus address of the outbound packet register includes a first bus address, and the device includes an inbound packet register, having a second bus address in the address space of the bus and adapted to receive and store an inbound network address header and payload data from an inbound packet received over the network by the network interface adapter, wherein the inbound address header and payload data are read by the master device from the inbound packet register in a further one or more of the bus cycles received by the bus interface adapter. Most preferably, the outbound network address header specifies a local identifier associated with the network interface adapter as a sending address of the outbound packet, and wherein responsive to the sending address, the target device addresses the inbound packet to the local identifier.
In a preferred embodiment, the bus includes a Peripheral Component Interface (PCI) bus, and the network includes an InfiniBand fabric. Preferably, the bus interface adapter includes a channel adapter, and the network interface adapter includes a switch.
There is also provided, in accordance with a preferred embodiment of the present invention, a bridge device for coupling a packet network to a parallel bus, the device including:
a network interface adapter, coupled to the network so as to receive over the network an inbound packet containing bus cycle information, including a bus address and command code;
a bus cycle register, adapted to receive and store the bus cycle information; and
a bus interface adapter, coupled to the parallel bus so as to generate a bus cycle of a type specified by the command code, and addressed to the bus address, as specified in the information stored in the bus cycle register.
Preferably, the bus cycle register is associated with a virtual address on the network, to which a host device on the network writes the bus cycle information by means of the inbound packet.
Additionally or alternatively, the bus cycle information further includes data to be written to the bus address.
Preferably, the device includes a register status indicator, which is set to indicate that all of the bus cycle information has been written to the bus cycle register, wherein the bus interface adapter drives the bus cycle responsive to setting the status indicator.
There is additionally provided, in accordance with a preferred embodiment of the present invention, a method for coupling a parallel bus to a packet network, including:
receiving one or more bus cycles from a master device on the bus, addressed to a bus address of an outbound packet register in an address space of the bus, the cycles conveying over the bus an outbound network address header and payload data;
incorporating the outbound network address header and payload data in an outbound packet; and
transmitting the outbound packet over the network to a target device on the network specified by the network address header.
There is further provided, in accordance with a preferred embodiment of the present invention, a method for coupling a packet network to a parallel bus, including:
receiving over the network an inbound packet containing bus cycle information, including a bus address and command code; and
generating a bus cycle of a type specified by the command code and addressed to the bus address, as specified by the bus cycle information.